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1.  Introduction 


Hyperspectral  imaging  has  found  wide  usage  in  the  remote  sensor  community  and  more  recently,  we 
have  seen  a  branching  out  into  the  commercial  (crop  health  monitoring  and  food  management  for 
example)  and  the  medical  arena.  One  exciting  medical  research  application  is  the  use  of 
hyperspectral  techniques  to  map  functional  areas  of  the  brain.  In  military  applications,  hyperspectral 
data  can  detect  chemical  and  biological  agents,  penetrate  foliage  to  detect  troops  and  vehicles,  lead 
to  the  creation  of  improved,  smart  missile  seekers  and  serve  as  a  countermeasure  against  ground 
targets,  employing  camouflage  and  concealment  techniques.  Segmentation  of  HSI  data  is  normally 
based  on  discrimination  between  materials  using  the  spectral  signature  in  thermal  emissions  or  solar 
reflectance  [3].  It  is  also  important  to  reduce  the  data  volume  and  dimensionality  without  loss  of 
discrimination.  Many  methods  have  been  attempted  in  the  past;  Harsanyi  and  Chang  [4]  used  an 
orthogonal  subspace  projection  approach,  projecting  each  HSI  pixel  vector  onto  a  subspace  that  is 
orthogonal  to  undesired  signatures.  In  this  report,  we  will  present  results  using  a  similar  and  simpler 
technique,  generating  the  “principal”  components  for  unknown  HSI  pixel  vectors  using  only  target 
eigenvectors  followed  by  segmentation  of  the  image. 

H.  Kwon  et  al.  [3]  used  mean  spectral  band  curves  of  HSI  pixels  for  different  materials  and 
developed  an  adaptive  segmentation  technique  with  reduced  computational  cost.  Rauss  et  al.  [5] 
employed  another  adaptive  technique,  genetic  programming  that  had  two  advantages;  first,  no 
assumptions  were  needed  about  the  character  of  the  HSI  data  and  secondly,  genetic  programming 
results  elaborate  which  spectral  bands  are  most  useful  for  discrimination.  The  joint  band 
prioritization  and  decorrelation  approach  by  Chang  et  al.,  demonstrated  that  in  some  cases,  over  90% 
of  the  total  HSI  spectral  bands  could  be  removed  without  loss  in  discrimination  [6]. 

I. 1  Hyperspectral  Imagery  and  Preprocessing 

One  tactical  application  that  is  our  primary  concern  for  investigation  is  the  detection  of  low-contrast 
thermal  targets  using  the  LWIR  region  extending  from  7.5  pm  to  13.7  pm. 

In  Figure  1 ,  we  have  an  LWIR  broadband  image  of  interest.  This  image  is  the  summation  of  all 
radiance  values  across  the  spectral  range  for  each  pixel.  Images  like  this  example  were  used  to 
derive  a  feature  space  for  the  target,  tree  and  ground  radiometric  data  for  128  spectral  bands  (7.5  - 
13.7  pm)  of  interest  within  one  pixel. 
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Figure  1.  LWIR  Summation  of  RD7  image.  A  IM60  tank  is  close 
to  the  center. 


Earlier  work  in  our  group  and  others  [1-3,  5]  showed  that  it  was  useful  to  convert  the  radiometric 
SEBASS  data  to  apparent  temperature  using  the  relationship: 


Tapp  (  2c 

k  A  log  z —  +  1 

U5Q 

with  Q  in  Watts/  i  the  wavelength  and  (1) 

/  cm 

h ,  c ,  and  k  constants. 


One  can  readily  see  the  distinction  present  when  comparing  radiance  to  relative  temperature.  The 
plots  in  Figure  2  were  taken  from  the  same  pixel  location  for  a  target  of  interest.  The  slight 
depression  in  the  spectral  radiance  plot  near  9  pm  is  far  more  obvious  in  the  relative  temperature 
profile.  These  slight  differences  in  shape  characteristics  can  be  exploited  for  segmentation  and 
classification  of  materials  once  the  inherent  “black-body”  curve  is  removed  from  the  radiance 
measure,  thus  the  utility  of  conversion  to  relative  temperature. 


2 


Relative  Temperature  Rot  on  Target  Pixel 


Spectral  Radiance  Plot  on  Target  Pixel 


-  1  !i 

\}\  /'J 

■  '  i1 

\  t  !l; 

700 

\ 

680 

i  j  | 

«s 

ct 

e 

1  640 
£ 

s 

1  620 
1 

/  \ 

/ 

1 

/ 

\  ■ 

\i 

/ 

/ 

/ 

\j 

580 

Figure  2.  The  left  image  is  a  relative  temperature  plot  for  a  U.S.  target  pixel  and  on  the 
right,  the  radiance  plot  for  the  same  pixel  location. 


2.  Feature  Extraction 


We  can  consider  equation  (1),  a  feature  extraction  processing  step  with  the  resulting  relative 
temperature  profile  representing  the  new  feature  space.  Fundamentally,  the  feature  extraction 
process  involves  choosing  a  set  of  features  for  a  class  of  patterns  that  will  exhibit  and  maintain  class 
separability  under  the  constraint  of  some  criterion  function.  Typically,  feature  extraction  is 
considered  a  mapping  from  an  w-dimensional  space,  to  a  smaller  subspace  n  and  here  we  can 
consider  the  removal  of  the  redundant  feature,  the  “black-body”  curve  as  a  reduced  feature  space. 
Further  reduction  in  the  dimension  of  the  feature  space  can  be  accomplished  by  various  means. 

Consider  the  following  set  of  relative  temperature  curves  for  the  various  classes  we  will  investigate. 

In  Figure  3,  we  see  some  of  the  variety  in  the  relative  temperature  profiles  for  each  class.  Take 
notice  of  the  sharp  depression  in  the  U.S.  target  and  exposed  ground  class,  or  the  broad  sweeping 
depression,  slightly  shifted  toward  longer  wavelengths  in  the  foreign  target  case. 
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Figure  3.  The  relative  temperature  plots  for  the  6  classes  of  target/clutter  considered  in  the  paper. 
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2.1  Broadband,  Narrowband  and  Sub-Band  Features 

One  feature  extraction  and  segmentation  method  to  consider  is  using  comparisons  of  features 
derived  from  narrow-band  and/or  broadband  ratio  and  difference  characteristics  in  the  relative 
temperature  profiles.  Broadband  ratios,  chosen  carefully,  should  yield  a  feature  or  set  of  features 
that  allow  discrimination  using  simple  techniques  for  the  target  and  foreign  target  and  to  some 
degree,  discriminating  exposed  ground  in  the  presence  of  the  remainder  of  the  classes.  Exposed 
ground  has  been  a  difficult  issue  for  segmentation  because  it  closely  resembles  target  signatures. 
Take  notice  in  Figure  3  of  the  foreign  target  spectrum  and  the  broad  depression  from  8  to  1 1  pm,  and 
compare  this  to  the  narrow  depression  exhibited  by  the  U.S.  target.  A  broadband  ratio  could  be  used 
in  this  case.  As  an  example,  calculate  the  ratio  of  the  relative  temperature  from  8. 5-9. 5  pm  and  9.5- 
10.5  pm.  Although  as  can  be  seen,  the  “exposed”  ground  will  confuse  the  issue  with  the  target  if 
this  were  the  only  feature  considered.  Further  feature  extraction  can  be  accomplished  by  using 
narrow-band  ratios  or  narrow-band  differences  [7].  Consider  the  exposed  ground  spectral  plot;  in  all 
cases,  we  see  a  pronounced  depression  in  the  area  slightly  >8  pm.  A  good  narrow-band  difference 
feature  is  the  difference  between  relative  temperature  profiles  at  8.5788  and  8.2706  pm.  This 
difference  can  be  far  more  pronounced  for  other  exposed  ground  pixels.  Camouflage  HSI  signatures 
present  another  problem.  Often,  we  see  mixed  pixels,  which  do  not  lend  themselves  to  segmentation 
although  some  detection  is  possible.  Simple  pixel  identification  is  probably  not  sufficient  for 
segmentation  of  camouflage  targets.  Sub-banding  is  simply  selecting  regions  of  the  relative 
temperature  profiles  for  segmentation  purposes  that  exhibit  less  redundancy  over  the  range  of 
wavelengths.  This  has  proven  to  be  useful  in  past  research  as  well  [1]  and  may  give  some  immunity 
to  diurnal  variations.  We  will  revisit  this  topic  later  in  the  paper. 

2.2  Principal  Component  Analysis 

Another  feature  extraction  method  that  holds  promise  is  principal  component  analysis  (PCA)  and  has 
been  used  for  band  selection  of  multi-spectral  imagery  [8]  in  the  past.  PCA  is  also  regarded  as  a 
decorrelation  technique  widely  used  in  data  compression  and  interpretation.  This  technique  exhibits 
a  high  degree  of  energy  compaction;  it  transforms  the  original  feature  space  into  an  uncorrelated 
space,  thus,  you  gain  the  advantage  of  reduced  complexity  in  your  segmentation/classification 
architecture  due  to  the  reduced  dimension  of  the  feature  space.  Principal  components  are  derived 
from  the  following  set  of  relationships:  Let  u ,  defined  as 

U  =  [?/(&>, ),  u(co2 ), . u(coN  )]7 ,  (2) 

be  an  N  x  1  random  input  vector  and  assume  zero  mean  without  loss  of  generality.  Let  R ,  be  the  N 
x  N  correlation  matrix  of  the  data  with  eigenvalues  ^,A2,...,AN .  The  k  principal  components  are 
defined  by  the  linear  transformation, 

C  =  O7  u ,  (3) 
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where  C  =  [c, , c2  ,...,ck^  is  a  k  x  1  principal  component  matrix,  and  <£>  is  an  N  x  k  matrix,  with 
columns  corresponding  to  the  k  eigenvectors  for  the  k  largest  eigenvalues  of  R .  We  can  choose  the 
value  for  k  arbitrarily,  where  k  is  significantly  smaller  than  N,  thus  reducing  the  feature  space  and  in 
some  cases,  retaining  separability. 

We  hope  to  exploit  PCA  with  an  appropriate  classifier  architecture  in  the  segmentation  of  the 
various  classes  and  targets.  Figure  4  includes  a  plot  of  the  magnitudes  for  the  first  six  eigenvalues 
associated  with  the  target,  tree  and  ground  pixels  from  the  HSI  training  set.  The  magnitudes  of  the 
2nd,  3rd  and  4th  for  the  three  classes  is  of  special  importance.  The  projection  of  a  test  pixel 
(principal  components)  onto  the  eigenspace  of  the  target  could  be  beneficial. 
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Figure  4.  The  relative  temperature  plot  on  the  left  displays  U.S.  target  in  solid  line,  the  tree  clutter  in 
dot/dashed  line,  and  the  ground/grass  clutter  in  dashed  line. 

The  LWIR  spectrums,  similar  to  those  in  Figure  4  are  projected  into  the  target  eigensubspace  with 
the  resulting  projections  passed  into  neural  network  for  discrimination. 

In  Figure  5,  we  see  eight  feature  components  being  formed  by  the  process  and  then  passed  to  the 
trained  neural  network  for  discrimination. 
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2nd  Eig^rtvcctor 


Figure  5.  The  projection  process  using  eigenvectors  derived  from  target 


3.  Segmentation  Techniques 


We  have  addressed  the  feature  extraction  and  optimization  problem  somewhat,  but  the  caveat  to  the 
problem  at  hand  is  in  the  selection  of  an  appropriate  classifier  topology  for  efficient  segmentation. 
Excellent  features  would  be  amenable  to  any  type  of  classifier  topology  [9],  but  the  multi-dimension 
nature  of  the  HSI  space  may  require  one  to  consider  classifiers  that  can  parse  the  feature  space 
efficiently.  We  will  not  consider  parametric  architectures,  although  it  may  prove  that  a  model  with 
statistical  parameters  fitting  the  HSI  space  has  exceptional  performance.  This  is  an  avenue  we  may 
take  in  the  future.  Non-parametric  and  boundary  decisions  classifier  topologies  are  able  to  model 
statistical  distributions  that  are  highly  non-Gaussian  and/or  multi-modal  in  nature,  and  past  work  has 
pursued  these  somewhat.  We  will  employ  these  architectures. 

In  coarse  observation  of  the  relative  temperature  profiles  in  Figure  6,  distinctions  are  readily 
apparent,  which  may  allow  very  simple  segmentation  techniques  to  be  employed.  The  integration  of 
specific  spectral  regions  could  prove  valuable,  also,  distance  metrics  should  be  able  to  take 
advantage  of  the  apparent  differences  as  seen  when  comparing  target  profiles  to  grass  and  ground  in 
the  regions  from  8.0  to  1 0.0  pm  and  the  region  from  1 1 .0  to  12.6  pm. 
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Mean  IWIR  Hyperspectral  Profile  of  Target  Mean  LWI R  Hyperspectral  Profile  for  Forested  Area 


IWIR  Spectrum  in  microns  LWIR  Spectrum  in  microns 


Mean  LWIR  Hyperspectral  Profile  of  Ground/Grass 


LWIR  Spectrum  in  mtcrons 


Figure  6.  From  left  to  right,  target,  tree  and  grass/ground  mean  relative  temperature  plots. 
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3.1  Simple  Segmentation  Algorithms 

Euclidean  distance  between  two  patterns,  X  and  Z , 

D  =  |X-Z||,  (4) 


is  one  measure  of  similarity. 

Figure  7  demonstrates  a  full-band  segmentation  result  using  an  Euclidean  distance  measure. 


Azimuth  Azimuth 


Figure  7.  On  the  right,  we  have  a  segmentation  result  using  all  128  bands  for  the  distance  metric. 

In  comparing  the  summation  image  and  the  segmented  image  of  Figure  7,  our  two  class  (target, 
background)  segmentation  was  quite  effective  and  preliminary  results  are  very  promising.  However, 
the  problem  becomes  much  more  difficult  when  considering  the  various  camouflage,  concealment 
and  deception  methods,  diurnal  effects  and  simple  placement  of  the  target  within  a  tree  line.  Also, 
the  ground  pixel  relative  temperature  profiles  are  richer  in  diversity  and  required  an  expansion  of  our 
training  database.  These  factors  dictate  the  need  to  consider  alternative  segmentation  methods. 

Several  useful  distance  measures  can  be  of  benefit  when  considering  statistical  properties  of  the  HSI 
patterns.  The  Mahalanobis  distance  is  given  by 

£>  =  (X-M)7'  CT1  (X-M),  (5) 

with  C  ;  the  covariance  matrix  of  the  pattern  population  M  as  the  mean  vector,  and  X  as  the 
unknown  pattern,  have  been  useful  in  the  past  but  their  performance  relies  on  the  underlying  class 
distribution  [7]  being  nearly  multivariate  Gaussian.  Also,  the  inverse  of  the  covariance  matrix  must 
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be  estimated  from  the  eigendecomposition;  direct  estimation  is  impossible  due  to  the  “ill- 
conditioned”  nature  of  covariance  matrix.  Another  measure  that  has  been  useful  in  taxonomy  and 
information  retrieval  is  the  so-called  “Tanimoto  measure,”  given  by 


S(X,Z) 


X7Z 

xrx  +  z7z-x7'z 


(6) 


with  known  pattern  vector  Z ,  is  maximized  when  the  cross-correlation  is  maximum.  Non-metric 
similarity  estimates  also  exist  which  measure  the  orientation  in  multidimensional  space  and  are  of 
benefit  when  classes  develop  along  principal  axes  [10].  One  such  method  is  the  cosine  of  the  angle 
between  the  unknown  vector  X  and  the  pattern  vector  Z  given  by 


S(X,  Z) 


X7'  z 

MPT 


(7) 


and  is  maximized  when  the  unknown  vector  and  pattern  vector  are  oriented  in  the  same  direction 
with  respect  to  the  origin. 


3.2  Gaussian  Mixture 

A  powerful  non-parametric  classifier  we  adopted  for  some  preliminary  test  was  the  Gaussian 
mixture  architecture.  The  Gaussian  mixture  classifier  is  a  refinement  of  the  multivariate  Gaussian 
technique  and  is  considered  a  nonparametric  technique  because  one  will  use  the  data  to  generate  the 
mixed  Gaussians.  In  this  case,  the  probability  distributions  are  modeled  as  a  weighted  set  of 
Gaussians.  The  conditional  distributions  are  given  by 

Ng 

p(X/Cj)='£dwkGk  (8) 

k=\ 

with  w/t ,  the  weight  of  the  k-th  Gaussian  G ^  and  the  weights  sum  to  one.  The  Gaussians  are  given 
by 

Gj= - A - -exp  (-Ux-Mk)TRk-\x~Mk))  (9) 

nx/AhV2 

with  Rfr  ,  the  covariance  matrix  for  the  kth  Gaussian,  and  M ^  as  the  corresponding  mean  vector. 

Training  is  accomplished  using  an  iterative  procedure  known  as  the  estimate-maximize  algorithm 
[11],  which  maximizes  the  likelihood  of  the  training  set  generated  by  the  probability  distribution 
function.  The  parameters  estimated  are  simply  the  weights,  mean  vectors  and  covariance.  See 
references  [12]  and  [13]  for  the  training  procedure.  Table  1  has  a  preliminary  classification  result 
using  a  Gaussian  mixture  architecture.  In  future  work,  we  will  investigate  this  architecture  further  as 
well  as  other  techniques. 
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Table  1.  The  values  in  the  table  represent  the  percentage  of 
pixels  classified  using  cross-validation  with  correct 
classification  along  the  main  diagonal. 


Gaussian  Mixture  (%) 


target 

tree 

ground 

target 

86 

0 

0 

tree 

14 

99 

10 

ground 

0 

1 

90 

3.3  Boundary  Decision  Architectures 

There  are  several  boundary  decision  architectures  available;  for  example,  the  simplest  and  most 
often  used  is  the  back-propagation  neural  network  (BPNN),  which  allows  for  separation  of  complex 
hyperboundaries  depending  on  the  number  and  size  of  the  hidden  layers.  We  have  employed  the 
BPNN  with  an  adaptive  learning  rate  that  allows  fine-grain  adjustments  during  training.  Smoothing 
is  also  incorporated  and  allows  the  control  of  weight  adjustment  based  on  the  past  values  of  gradient 
descent  and  can  prevent  the  training  process  from  terminating  in  shallow  local  minimum  [14].  For 
target  identification  and  segmentation,  the  BPNN  can  provide  both  a  robust  classifier  and  a  measure 
of  your  confidence  in  the  classification  decision.  BPNNs’  derive  their  computational  power  from 
the  parallel-distributed  structure  and  the  ability  to  learn  and  adapt.  Specifics  governing  the  neural 
network  training  flow  are  beyond  the  scope  of  this  paper.  We  will  present  several  results  using  the 
BPNN  for  segmentation  in  later  sections  of  the  paper. 
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4.  Hyperspectral  Database 


SEBASS  HSI  hypercubes  were  radiometrically  calibrated  [7]  and  the  resulting  3-dimensional  (3-D) 
cubes  were  used  for  developing  our  database.  The  calibrated  hypercubes  are  128(A)  x  131  x  128 
with  128  individual  bands  ranging  from  7.5466  to  13.6772  pm  and  bandwidths  from  0.0691  to 
0.0368  pm.  The  relative  temperature  training  database  we  constructed  consists  of  U.S.  and  foreign 
target  pixels  where  both  open  terrain  and  CCD  targets  are  represented.  The  tree,  ground  and 
exposed  ground  pixels  were  selected  arbitrarily  but  with  great  care  in  parsing  them  after  initial 
selection  to  remove  outliers  or  what  may  be  considered  as  mixed  pixels.  This  procedure  though  is 
very  subjective  and  represents  a  problem  for  the  camouflaged  cases.  The  training  pixels  of  all 
classes  also  represent  a  cross  section  of  the  HSI  images  from  5:00  am  to  8:00  pm;  however,  a  set  of 
hypercubes  where  “temperature  inversion”  is  observed  was  not  included. 

4.1  Ground  Truth 

In  order  to  render  the  images  for  viewing,  a  summation  of  all  radiance  values  across  the  LWIR 
spectrum  was  performed  (Figure  8). 

These  are  the  representative  scenes  we  have  considered  in  our  work.  Both  are  very  rich  in  diversity 
with  respect  to  the  relative  temperature  profiles  for  various  targets.  In  Figure  8,  both  scenes  have 
several  targets  of  interest  and  clutter.  Target  and  type  are  located  at  the  following  positions  for 
RD7XXXX  files. 


RD7  01  Band  Summation 


20  40  60  60  100  120 

Azimuth 


RDM  01  Band  Summation 


Azimuth 


Figure  8.  Summation  images  for  RD7  and  RDM  cases. 
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Azimuth 

Elevation 

Target 

X  72 

Y  64 

M60  NETTING  NONE 

X  84 

Y  55 

T72  NETTING  NONE 

X  122 

Y  53 

ZIL131  NETTING  NONE 

X  28 

Y  68 

BTR70  NETTING  NONE 

The  M60  is  readily  apparent  in  the  center  of  the  image,  see  Fig.  8,  left-hand  side. 

The  RDMXXXX  files  have  targets  located  as  follows  but  with  large  tracts  of  exposed  ground  filling 
several  areas  of  the  image  as  well. 


Azimuth 

Elevation 

Target 

X  2 

Y  63 

M2  NETTING  NONE 

X  54 

Y  59 

Ml  NETTING 

LCSSWOOD 

The  M2  is  near  the  left  border  and  the  summation  image  indicates  something  unusual  at  this 
position. 

4.2  Characteristic  Hyperspectral  Imagery 

The  following  images  in  Figure  9  exhibit  several  of  the  characteristic  renderings  for  the  two  regions 
of  interest  in  this  paper.  One  should  compare  the  band  summation  directly  with  the  band-subtraction 
results,  which  was  shown  to  be  useful  in  detecting  some  CCD  targets  by  various  researchers  [2,  7]. 
The  band-subtraction  was  performed  by  subtracting  the  radiance  values  at  10.77  pm  and  9.97  pm. 
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Bifid  Sutstritflyi 


Figure  9.  Band  subtraction  and  band  summation  images  for  RD7  and  RDM  eases. 


On  close  inspection  of  the  RDM  images,  one  can  see  the  camouflaged  target  sitting  at  elevation  59 
and  azimuth  54.  Band  summation  fails  to  distinguish  any  target  presence  and  is  similar  to  what  the 
naked  eye  may  determine;  however,  band-subtraction  suggests  the  presence  of  something  unusual 
sitting  in  the  group  of  trees.  Band  subtraction  suggests  that  other  narrow  band  features  may  be  of 
utility  in  discriminating  various  classes.  A  goal  is  to  further  bring  out  the  features  in  the  RDM 
image  group,  especially  the  camouflage  target. 


5.  Segmentation  Results 


We  present  the  stepwise  segmentation  on  full-band,  relative  temperature  images  using  the  BPNN  as 
our  choice  for  classifier  architecture.  Each  subsection  will  be  devoted  to  particular  subset  of  classes. 

5.1  Segmentation  of  3  Classes  Using  BPNN 

The  8  images  of  Figure  10  demonstrate  the  result  of  segmentation  for  the  3-class  case  using  the 
trained  BPNN.  The  3  class  case  only  includes  tree,  ground/grass  and  U.S.  target  (Figure  3)  We  see 
that  we  can  clearly  discriminate  the  target  from  the  background  in  the  RD7  case,  but  the  exposed 
ground  present  in  the  RDM  case  greatly  obscures  the  result.  The  relative  temperature  profiles  for 
exposed  ground  are  very  similar  to  the  U.S.  target  pixels  chosen  for  training.  In  some  instances,  we 
detect  something  in  the  tree  line  for  RD7,  and  we  can  distinguish  the  M2  target  at  azimuth  =  2  and 
elevation  =  63  in  the  RDM  cases,  but  the  camouflaged  Ml  is  completely  invisible  to  the 
segmentation,  and  the  resulting  confusion  of  the  exposed  ground  must  be  addressed.  The  full 
spectrum  performs  well  for  the  simple  case  but  suffers  in  the  camouflaged  case.  In  RDM  23,  we  can 
detect  a  portion  of  the  target  but  the  “false”  alarms  are  too  great.  The  RD715  and  RDM  14  results  are 
due  to  “temperature  inversion”  and  we  will  see  several  cases  of  this  problem  in  other  segmentation 
results. 
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Figure  10.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  only  the  3 
classes,  U.S.  target,  tree  and  grass/ground. 


5.2  Segmentation  4  Class  Using  BPNN 

Let  us  add  the  exposed  ground  class  to  the  problem.  What  follows  in  Figure  1 1  are  the  results  for  the 
4-class  segmentation. 


RD701  resull.  Neural  Net  trained  with  4  dasses 


RDMOI  result.  Neural  Net  trained  with  4  classes 


20  40  60  80  100  120 

Azimuth 


RDM  14  result.  Neural  Net  trained  with  4  classes 


RDM23  result.  Neural  Net  trained  with  4  classes 


Azimuth 


Figure  11.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  only  the  4  classes, 
U.S.  target,  tree,  exposed  ground  and  grass/ground. 
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We  see  that  the  addition  of  the  exposed  ground  class  has  clarified  the  issue.  In  these  renderings,  the 
exposed  ground  is  dark  gray  to  black;  however,  some  pixels  still  are  not  classified  in  these  images. 
We  can  clearly  see  the  M2  target  in  the  RDM  images  but  the  hidden  targets  in  the  tree  line,  the 
foreign  target  and  camouflaged  target  are  still  indistinguishable. 

5.3  Segmentation  5  Class  Using  BPNN 

Let  us  add  the  foreign  target  class  to  the  4  class  problem.  What  follows  in  Figure  12  are  the  results 
for  the  5  class  segmentation. 

Now  we  have  complicated  the  issue  further,  adding  a  5th  class  in  Figure  12,  with  the  addition  of  a 
foreign  target. 
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RDM01  result,  Neural  Net  trained  with  5  classes,  foreign  Included 


Figure  12.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  5  classes,  U.S. 
target,  foreign  target,  tree,  exposed  ground  and  grass/ground. 
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We  see  with  the  addition  of  the  foreign  target  (see  Figure  3),  the  trained  BPNN  with  5  classes  begins 
to  exhibit  some  degradation  in  performance  in  segmenting  the  U.S.  targets.  Compare  the  3  and  4 
class  cases  for  the  M60  (RD7XXXX)  at  location  x  =  72  and  y  =  64  with  the  5  class  cases.  The 
temperature  inversion  and  5  class  problem  have  completely  degraded  the  performance  as  seen  in 
RD715.  We  are  able  though  to  detect  several  targets  in  the  RD7  case,  especially  the  late  evening 
images  RD723.  The  region  near  x  =  50  and  y  =  30  is  a  mystery  but  these  pixels  on  closer 
examination  have  some  similarity  to  the  selected  pixels  for  foreign  target. 

5.4  Segmentation  of  4  Class,  Foreign  Only 

Let  us  return  to  a  4  class  problem  with  the  foreign  target  included  and  without  U.S  target. 

We  see  that  in  the  foreign  target  case  in  Figure  13,  we  haven’t  gained  anything  with  regard  to 
segmentation  and  in  fact  may  have  lost  some  robustness.  The  figure  below  is  RDM23,  a  hypercube 
collected  at  -10:00  pm  in  the  evening. 


RD701  result,  Neural  Net  trained  with  4  classes 


RDM01  result.  Neural  Net  trained  with  4  classes 


Figure  13.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  4  classes,  foreign  target,  tree, 
exposed  ground  and  grass/ground. 

Unlike  the  other  trained  networks,  in  Figure  14  we  see  difficulty  in  distinguishing  the  tree  line  from 
grass,  which  may  be  partly  due  to  the  region  falling  in  shadow  over  the  last  few  hours  but  moreso, 
the  partitioning  of  the  trained  network  is  now  sensitive  to  these  boundaries.  You  can  see  the  darker 
gray  regions  that  fill  the  area  between  the  true  tree  line  and  the  grass/ground  pixels  as  well  as 
problems  with  the  exposed  ground  area  near  the  bottom  of  the  image. 
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RD723  result.  Neural  Net  trained  with  4  classes 


Figure  14.  Degraded  segmentation  performance  with  foreign 
target  4  class  problem. 

5.5  Segmentation  Comparison  of  4  Class  And  5  Class,  U.S.  and  Camouflaged  Target 

Let  us  look  at  the  5  class  case  segmentation  results  in  Figure  1 5  where  we  have  added  camouflage 
and  U.S.  target  to  the  training  procedure  and  compare  to  the  4  class  case  with  only  the  camouflage 
vehicle  added. 


A2imi*h 


AarmXh 


RDM23  result.  Neural  Nel  trained  with  4  classes 


RDM23  result.  Neural  Net  trained  with  5  classes  camouflaged  included 


RDM  23  Band  Subtraction 


Arimuth 


Figure  15.  On  the  left,  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  4 
classes,  camouflaged  target,  tree,  exposed  ground,  and  grass/ground.  On  the  right,  RD7  and 
RDM  segmentation  results  using  a  BPNN,  trained  with  5  classes,  U.S.  target,  camouflaged 
target,  tree,  exposed  ground,  and  grass/ground.  Band-subtraction  images  are  at  the  bottom  of 
figure  for  comparison. 
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Here  we  see  nearly  identical  results  with  the  4  or  5  class  trained  network  and  a  great  improvement 
over  the  band-subtraction  method  for  detecting  the  camouflage  vehicle.  A  segmentation  result  with 
an  algorithm  using  spatial  techniques  could  prove  very  useful.  This  result  is  very  exciting 
considering  the  difficulty  in  selecting  proper  training  pixels  for  the  camouflage  training.  We  will 
elaborate  on  this  in  the  conclusion  section. 

One  more  interesting  result  can  be  seen  in  Figure  16  with  the  4  class,  camouflage  trained  networks. 
The  RD7  images  have  targets  embedded  in  the  tree  line  and,  therefore,  several  mixed  pixels  exist  in 
the  image  near  these  targets.  Compare  Figure  16  with  the  RD7  case  of  Figure  12. 


RD701  result.  Neural  Net  trained  with  4  classes 
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RD701  result.  Neural  Net  trained  with  5  classes,  camouflaged  Included 


Figure  16.  RD7  segmentation  results  using  a  BPNN,  trained  with  4  classes,  camouflaged  target, 
tree,  exposed  ground,  and  grass/ground,  demonstrating  the  mixed  pixel  segmentation. 


These  results  demonstrate  the  mixed  pixel  nature  of  the  camouflage.  Observe  the  target  located  at  x 
=  122  and  y  =  53.  We  have  not  been  able  to  detect  this  target  using  either  U.S.  or  foreign  4  class 
trained  networks  but  in  the  camouflage  case  it  clearly  shows  up.  Also,  the  pixels  near  the  hidden 
Russian  T-72  tank  at  x  =  84  and  y  =  55  are  not  the  same  that  we  detected  earlier  using  the  foreign 
case  (Figure  13). 

5.6  Segmentation  of  3  Class,  Clutter  Only 

Figure  17  again  demonstrates  the  great  similarity  between  the  target  and  exposed  ground  with 
respect  to  the  parsing  of  the  feature  space.  Here,  only  the  clutter  classes  were  used  and  you  can 
readily  see  the  targets  despite  a  threshold  applied  to  the  output  neuron  results.  In  the  RDM  case,  you 
see  the  M2  clearly  at  x  =  2,  y  =  63. 
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RD701  resun,  Neural  Net  trained  with  3  clutter  ciast 


Ft  DM01  result,  Neural  Net  trained  with  3  duller  classes 
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RD723  result  Neural  Net  trained  with  3  clutter  classes 


RDM23  resu*.  Neural  Net  Ira  ned  wth  3  clutter  cla: 


Figure  17.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  3  classes,  tree,  exposed 
ground,  and  grass/ground. 


6.  Segmentation  Based  on  Distance  Metric 


The  following  sections  are  examples  of  other  feature  methods  applied  in  the  3  class  problem  with 
hopes  of  simplifying  and/or  improving  segmentation.  Figure  1 8  demonstrates  several  sub-band 
segmentation  results  derived  with  the  distance  metric.  The  sub-band  region  was  extracted  from  8.7 
to  11.3  pm  in  the  relative  temperature  profile  and  gave  similar  results  to  the  distance  metrics  for  full 
band  detection  and  segmentation  result  using  only  the  distance  metric  given  by  equation  (2). 
Compare  the  RD7  result  with  Figure  7. 
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RD7  01  Segmentation  Using  Distance  Metric 


RD7  15  Segmentation  Using  Distance  Metric 
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RD7  1 9  Segmentation  Using  Distance  Metric 
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RDM  01  Segmentation  Using  Distance  Metric 
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RDM  14  Segmentation  Using  Distance  Metric 


Azimuth 


RDM  23  Segmentation  Using  Distance  Metric 


Azimuth 


Figure  18.  RD7  and  RDM  segmentation  results  using  a  distance  metric  trained  with  3  classes,  U.S. 
target,  tree,  and  grass/ground. 


We  see  the  target  clearly  in  the  RD7  case  as  was  expected  but  cannot  detect  the  target  in  the  RDM 
case.  You  can  see  the  camouflaged  target  in  the  RDM  23,  only  in  the  10:00  pm  data  collection,  but 
again,  the  presence  of  the  “target-like”  exposed  ground  class  confuses  the  issue.  Clearly,  the 
detrimental  effect  of  the  feature’s  sensitivity  to  the  diurnal  cycle  resulted  in  the  RD719  image. 
Typically,  a  sub-band  approach  exhibits  some  immunity  to  the  diurnal  changes  rather  than  using  the 
entire  spectrum  for  the  simple  distance  metric. 


7.  Segmentation  Using  Broadband  and  Narrow-Band  Features 


The  broadband  feature  that  was  selected  uses  the  wavelengths  from  8  to  1 1  pm.  An  absolute  value 
of  the  ratio  of  the  sum  of  normalized  relative  temperature  values  between  8  and  9.5  pm  and  9.5  pm 
to  1 1.0  pm  is  the  feature.  The  narrow  band  features  are  the  differences  between  a  relative 
temperature  at  8.5788  and  8.2706  pm  as  well  as  at  8.9920  and  8.8174  pm.  These  features  were 
selected  “ad  hoc,”  by  looking  over  several  HSI  plots  for  various  classes  and  by  no  means  have  we 
exhausted  the  possible  choices. 

In  Figure  19,  we  can  clearly  see  many  of  the  targets  in  the  simple  3  class,  3-feature  case.  What  is 
very  striking  is  the  target  at  x  =  28  and  y  =  68  in  the  RD723  image,  a  BTR70  that  has  basically 
eluded  the  segmentation  techniques  used  earlier.  Again,  we  see  the  exposed  ground  confusing  the 
issue;  however  in  Figure  20,  where  we  have  added  the  fourth  class,  exposed  ground  and  collected  3 
features  for  each  class,  we  can  clearly  discriminate  the  target  and  exposed  ground.  This  is  an 
exciting  result  and  warrants  further  investigation  because  the  complexity  has  been  drastically 
reduced  with  the  3-feature  case. 
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RD701  result.  Neural  Net  trained  with  3  features  for  3  ctesse 


RDM01  result,  Neural  Net  trained  with  3  features  for  3  classes 


Figure  19.  RD7  and  RDM  segmentation  results  using  a  BPNN,  trained  with  3  classes,  U.S.  target, 
tree,  and  grass/ground.  Features  are  broadband  ratio  and  narrow-band  differences. 


|g|||gyg^ 

^ ji ^5  - ^*"'1^1 

RDM23  result.  Neural  Net  trained  with  3  features  for  4  classes 
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Figure  20.  RDM  Segmentation  result  using  a  BPNN,  trained 
with  4  classes,  U.S.  target,  tree,  exposed  ground,  and 
grass/ground.  Features  are  broadband  ratio  and  narrow- 
band  differences. 


The  narrow-band  difference  features  are  nearly  linear  separable  for  the  U.S.  target  and  exposed 
ground  class,  as  seen  in  Figure  21. 

Figure  21  is  a  (2-D)  of  the  two  narrow-band  features  for  exposed  ground  and  U.S.  target.  The 
“crosses”  are  the  2-D  points  for  the  narrow-band  features  for  exposed  ground,  almost  always 
positive  in  value,  whereas  the  “stars”  represent  the  2-D  location  of  the  U.S.  target  narrow-band 
features,  which  are  typically  negative  in  value. 
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Figure  21.  U.S.  target  and  exposed  ground  narrow  band  features  in  2-D. 
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8.  Principal  Components 


For  the  segmentation  of  full  spectrum  relative  temperature  components  and  the  PCA  components 
derived  front  these  same  sets.  A  BPNN  was  trained  using  one  hidden  node  for  each  input  node  in 
the  case  of  the  full  spectrum  components  and  various  values  for  the  total  hidden  nodes  were  tried 
with  the  PCA  features.  Training  was  performed  using  “epoch”-stopping  criteria.  Further  discussion 
of  the  networks  is  not  appropriate  for  this  technical  report,  but  see  reference  [14]  for  further 
discussions  on  neural  networks. 

The  images  in  Figure  22  are  characteristic  of  segmentation  using  the  BPNN’s  that  were  trained  with 
the  target  principal  components  and  the  projections  of  ground  and  tree  pixels  into  the  target’s 
eigenspace.  We  see  several  regions  that  produce  false  alarms  and  therefore  any  discrimination  is  lost. 
It  should  be  mentioned  that  these  images  are  the  result  of  the  target  output  node  class  only,  and  when 
applying  a  threshold  to  these  target  node  outputs,  clearer  images  can  be  obtained  with  a  small 
reduction  in  false  alarms.  False  alarms  are  in  part,  due  to  the  sensitivity  of  the  eigenprojection  to 
diurnal  changes.  A  shortcoming  of  this  eigenspace  projection  procedure  is  the  reliance  on  one  set  of 
eigenvectors.  Flowever,  an  expanded  projection  method  using  eigenvectors  for  the  various  classes  of 
interest  could  be  considered. 
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9.  Diurnal  Variation 


The  following  figures  are  examples  of  diurnal  variations  in  HSI  relative  temperature  profiles. 

As  one  can  readily  see  in  Figure  23,  diurnal  variations  in  relative  temperature  have  drastically 
changed  the  profiles  resulting  in  the  poor  performance  of  the  eigenspace  projections.  Compare  these 
with  the  means  of  the  training  set  in  Figure  3. 
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Figure  23.  Diurnal  variation  for  U.S.  target  and  tree  pixel  examples. 
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Figure  24  is  an  image  representative  of  the  trained  neural  networks’  entire  output  layer,  which  we 
have  seen  earlier  in  the  paper. 
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Figure  24.  RD7  segmentation  depicting  some  of  the  richness  in 
target/clutter  classes. 

This  image  shows  that  there  is  a  finer  class  structure  to  consider.  The  brightest  areas  are  target  and 
exposed  ground,  but  we  see  that  the  regions  of  exposed  ground  are  also  classified  in  some  cases  as 
the  ground/grass.  For  example,  look  at  [x,  y]  =  [10,  80]  in  Figure  24.  The  black  areas  represent 
shadowed  regions;  dark  gray  regions  are  ground/grass  and  trees  with  light  gray  regions  being 
predominantly  tree  locations.  Although  we  have  presented  results  earlier  for  far  more  classes,  we 
have  by  no  means  exhausted  the  partitioning  of  the  total  classes  of  clutter  and  target  (mixed  pixel)  in 
this  image  or  in  the  HSI  hypercubes.  These  data  were  collected  in  the  late  morning,  which  explains 
the  brighter  areas  of  exposed  earth  centered  around  [40,40]  incorrectly  classified  as  target. 
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10.  Conclusion 


The  features,  either  the  relative  temperature  profiles  themselves  and  those  derived  from  using  the 
relative  temperature  profiles  are  still  in  need  of  further  investigation.  The  broadband  and  narrow- 
band  features  could  be  extended  to  include  features  primarily  for  class  discrimination  of  the  U.S. 
target,  foreign,  and  perhaps  the  camouflaged  target  (mixed  pixels).  Candidate  features  include  a 
narrow-band  feature  derived  from  taking  the  difference  at  10  and  9  pm,  a  good  feature  for  U.S. 
targets  (unpublished  work),  and  another  narrow-band  feature  derived  from  the  difference  at  1 1  and 
10  pm,  which  should  prove  to  be  an  excellent  feature  for  discriminating  the  foreign  class.  A 
broadband  feature  suggests  itself  for  the  foreign  class  because  we  see  that  the  relative  temperature 
profile  “depression”  is  shifted  and  broader  for  foreign  classes.  In  the  foreign  target  case,  we  had 
great  difficulty  in  collecting  training  features,  there  simply  isn’t  enough  examples  in  the  database. 
The  camouflage  case  still  remains  the  most  difficult  although  results  are  promising.  Ground  truth  of 
our  current  database  is  insufficient  for  the  camouflage  training  collections  and  must  be  superior  to 
what  is  on  record  for  these  HSI  hypercubes.  The  closeness  of  the  mixed  pixel,  exposed  ground,  and 
camouflage  contribute  to  the  difficulty  in  selecting  a  “pure  camouflage”  pixel  for  training.  During 
future  data  collections  for  camouflage  pixels,  each  pixel  associated  with  the  camouflage  target  must 
be  recorded  in  the  ground  truth.  It  may  be  of  benefit  to  simply  lay  camouflage  netting  on  the  open 
ground  and  collect  data  at  the  same  time  you  gather  camouflaged  target  data.  Sub-banding  looks 
promising,  considering  that  only  50  spectral  bands  were  used  in  detecting  the  camouflaged  target  in 
RDM  23  case.  Also,  this  segmentation  using  the  sub-bands  was  performed  using  the  simple  distance 
metric.  A  low-complexity  classifier  architecture  coupled  with  an  effective,  reduced  feature  space  is 
the  primary  goal.  Sub-banding  and  narrow/broadband  features  fit  this  notion  and  hold  promise  as  a 
method  to  lend  immunity  to  diurnal  changes  as  well.  However,  the  full  spectrum  trained  nets  are 
superior  in  segmenting  the  images.  It  may  be  extremely  difficult  to  find  salient  features  that  can  give 
good  discrimination  over  the  entire  diurnal  cycle  and,  therefore,  future  efforts  will  be  focused  on  the 
early  morning  and  late  evening  cases.  PCA  may  serve  as  a  beneficial  feature  extraction  and 
compaction  method  once  we  restrict  the  empirical  work  to  the  cases  previously  mentioned,  but  at 
this  point,  there  is  no  advantage  to  report.  Only  using  the  2nd,  3rd  and  4th  “projection”  is  not  enough 
and  should  be  extended.  A  finer  parsing  of  the  various  classes  in  the  HSI  images,  for  example, 
including  the  class,  mixed  pixel  is  expected  to  improve  the  detection  process,  perhaps  considering 
the  presence  of  shadows  along  the  tree  line  as  a  specific  class  may  be  of  benefit  as  well. 
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The  segmentation  architecture  has  several  areas  for  improvement  and  exploration.  Adaptive 
techniques  and  algorithms  have  been  investigated  with  excellent  results  [1].  Recurrent  networks, 
feeding  the  output  layers  or  last  few  output  layer  results  back  to  the  input  layer,  would  bring  a  spatial 
reference  to  the  segmentation.  Codebooks  for  the  various  classes  could  be  constructed  contributing 
to  the  segmentation  options  and  utility.  Finally,  modeling  mixed  pixels  may  allow  one  to  artificially 
extend  the  camouflage  database,  with  the  hope  of  improving  segmentation. 
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